[backend:fix] fix same-kernel Python UDF registration and invocation MCDB-103581 by ShubMishra · Pull Request #156 · singlestore-labs/singlestoredb-python

ShubMishra · 2026-11-02T14:33:47Z

Summary

Disambiguate error 1347 using routine/table catalogs and SHOW AGGREGATES. Reject conflicting names before stopping the live server; keep non-replacing CREATE and published/other-session ownership checks.
Move interactive UDF HTTP serving off the notebook event loop so synchronous same-kernel SQL calls can complete. Preserve context, startup errors, re-registration, cancellation, and graceful shutdown.
Leave published UDF serving, notebook image pins, package version, and public APIs unchanged.

Test Plan

Staging (user-provided evidence): both UDFs registered successfully; same-notebook SELECT pulse.double_it_sh_test(2.0); returned 4.0. Cell duration: 529 ms.
Local: PYTHONASYNCIODEBUG=1 .venv/bin/python -m pytest -q -n 0 singlestoredb/tests/test_function_url.py singlestoredb/tests/test_udf_event_loop.py - 56 tests and 19 subtests passed on Python 3.11.12 / Uvicorn 0.34.3 against an isolated SingleStore 9.1.1 container. Covers actual 1347/1982/1050 responses, conflicting object types, case variants, schemas, and lifecycle behavior.
Checks: changed-file pre-commit checks (including flake8/mypy) and git diff --check passed. Full engine-protocol suites were not run locally.

The pure-Python fallback warning is expected with SINGLESTOREDB_BUILD_EXTENSION=0; it did not prevent registration or invocation.

Runtime Impact

Interactive apps gain one persistent HTTP serving thread/event loop, not a new server thread per query. Startup/shutdown coordination is outside the request path; existing UDF dispatch remains unchanged. The new loop can own an executor for synchronous UDF work. Latency, throughput, and memory overhead have not been benchmarked; the screenshot is a functional smoke test.

Deployment Plan

No shared staging/production image is changed by this PR or the manual runtime installation.
After SDK review/release and remaining staging lifecycle checks, update the Helios notebook image's SDK pin in a separate rollout.
Roll back the manual test by reinstalling the recorded prior SDK version/revision and restarting the kernel, or by recreating the disposable runtime from its unchanged image. A kernel restart alone does not restore the old package.
Python cannot forcibly stop arbitrary synchronous UDF code. Shutdown stops the serving loop without waiting indefinitely for such worker code; the existing limitation on cancelling the user code itself remains.

JIRA Issues

Resolves MCDB-103581.

Note

Medium Risk
Changes interactive UDF lifecycle (thread + second event loop) and SQL registration preflight; published UDF path is mostly unchanged but shutdown/task typing is shared.

Overview
Fixes interactive notebook Python UDF registration and same-kernel SELECT calls (MCDB-103581).

Interactive HTTP serving no longer runs Uvicorn on the notebook cell’s asyncio loop. When running_interactively is true, run_udf_app starts a ThreadedUvicornServer (dedicated thread + event loop, contextvars copied at thread start) while published workloads keep the in-loop AwaitableUvicornServer. shutdown_udf_app and startup failure paths were extended to await Future completions, forward shutdown to the worker loop, and join the thread so re-registration and cancellation do not leave listeners or workers behind.

Name conflict handling in _show_create_service_url: if SHOW CREATE FUNCTION fails with ER.WRONG_OBJECT (1347), the code now looks up information_schema and SHOW AGGREGATES to see whether the name is taken by a procedure, table, view, or aggregate, and raises a clear RuntimeError instead of treating the name as free—so preflight can reject conflicts before tearing down a live interactive server. Missing names still register with non-replacing CREATE.

Tests add interactive lifecycle coverage (same-kernel sync calls, context propagation, bind/retry, startup cancel), shared shutdown behavior for both server modes, and integration checks against real engine WRONG_OBJECT responses.

^{Reviewed by Cursor Bugbot for commit f590ddd. Bugbot is set up for automated code reviews on this repo. Configure here.}

Copilot

🟡 Changes recommended

The unshielded startup cleanup wait allows a second cancellation to invalidate the server’s shared completion Future, causing errors during completion and subsequent shutdown or re-registration.

1 open issue

Shield cleanup wait from cancellation to preserve completion Future · New

What changed in this PR

This PR fixes interactive Python UDF registration and allows synchronous SQL calls from the same notebook kernel by moving HTTP serving to a dedicated thread.

Changes:

Treat error 1347 from SHOW CREATE FUNCTION as a missing function while preserving non-replacing registration.
Add threaded interactive serving with startup and shutdown coordination; retain caller-loop serving for published UDFs.
Expand regression tests for registration, same-kernel invocation, context propagation, and server lifecycle.

File	Description
`singlestoredb/tests/test_function_url.py`	Adds registration and threaded-server regression tests.
`singlestoredb/functions/ext/function_url.py`	Recognizes `WRONG_OBJECT` during missing-function checks.
`singlestoredb/apps/_uvicorn_util.py`	Implements threaded serving and loop cleanup.
`singlestoredb/apps/_python_udfs.py`	Selects interactive threaded serving and coordinates its lifecycle.

_{Review effort: Balanced}

Copilot

🟢 Approval recommended

No active review issues block the interactive Python UDF registration and serving changes.

No open issues 🎉

1 resolved since last review

Shield cleanup wait from cancellation to preserve completion Future

_{Review effort: Balanced}

kesmit13

Threaded serving looks solid; one real gap in the 1347 handling, plus two small items.

Also: the shutdown_udf_app docstring still says to call it "on the same event loop as run_udf_app", which no longer applies to the threaded server.

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit ef78a7a. Configure here.}

Copilot

🟢 Approval recommended

No active review issues were identified in the interactive UDF registration and server lifecycle changes.

No open issues 🎉

_{Review effort: Balanced}

…MCDB-103581 Disambiguate SHOW CREATE error 1347 with catalog checks before stopping a live interactive server. Preserve non-replacing registration and reject procedure, aggregate, table, and view conflicts. Serve interactive UDF callbacks on a dedicated event-loop thread with safe startup, cancellation, and shutdown; retain published serving behavior. Cover registration and lifecycle regressions, including real-engine catalog and error-code checks.

Copilot

🟢 Approval recommended

The review identifies no active issues in the interactive Python UDF registration and serving changes.

No open issues 🎉

_{Review effort: Balanced}

ShubMishra deployed to Base November 2, 2026 14:33 — with GitHub Actions Active

ShubMishra marked this pull request as ready for review November 2, 2026 15:18

ShubMishra requested review from bharathts07, kesmit13, mgiannakopoulos, pmishchenko-ua and volodymyr-memsql as code owners November 2, 2026 15:18

ShubMishra requested a balanced review from Copilot November 2, 2026 15:20

Copilot started reviewing on behalf of ShubMishra November 2, 2026 15:21

View session

ShubMishra force-pushed the skumar/mcdb-103581-notebook-udf-loop branch from 66693d2 to 4cb5790 Compare November 2, 2026 15:26

ShubMishra deployed to Base November 2, 2026 15:26 — with GitHub Actions Active

Copilot AI reviewed Nov 2, 2026

View reviewed changes

Comment thread singlestoredb/apps/_python_udfs.py Resolved

ShubMishra force-pushed the skumar/mcdb-103581-notebook-udf-loop branch from 4cb5790 to 759c4dc Compare November 2, 2026 15:34

ShubMishra deployed to Base November 2, 2026 15:34 — with GitHub Actions Active

ShubMishra requested a balanced review from Copilot November 2, 2026 15:36

Copilot started reviewing on behalf of ShubMishra November 2, 2026 15:37

View session

Copilot AI reviewed Nov 2, 2026

View reviewed changes

kesmit13 reviewed Nov 2, 2026

View reviewed changes

Comment thread singlestoredb/functions/ext/function_url.py Resolved

Comment thread singlestoredb/tests/test_function_url.py Resolved

ShubMishra force-pushed the skumar/mcdb-103581-notebook-udf-loop branch from 759c4dc to ef78a7a Compare November 3, 2026 03:00

ShubMishra had a problem deploying to Base November 3, 2026 03:00 — with GitHub Actions Failure

cursor Bot reviewed Nov 3, 2026

View reviewed changes

Comment thread singlestoredb/functions/ext/asgi.py Resolved

ShubMishra force-pushed the skumar/mcdb-103581-notebook-udf-loop branch from ef78a7a to 334fcfe Compare November 3, 2026 04:01

ShubMishra deployed to Base November 3, 2026 04:01 — with GitHub Actions Active

ShubMishra force-pushed the skumar/mcdb-103581-notebook-udf-loop branch from 334fcfe to 88c4575 Compare November 3, 2026 04:21

ShubMishra had a problem deploying to Base November 3, 2026 04:21 — with GitHub Actions Failure

ShubMishra force-pushed the skumar/mcdb-103581-notebook-udf-loop branch from 88c4575 to 1ad4347 Compare November 3, 2026 06:09

ShubMishra had a problem deploying to Base November 3, 2026 06:09 — with GitHub Actions Failure

ShubMishra requested review from kesmit13 and a balanced review from Copilot November 4, 2026 06:44

Copilot started reviewing on behalf of ShubMishra November 4, 2026 06:45

View session

Copilot AI reviewed Nov 4, 2026

View reviewed changes

ShubMishra force-pushed the skumar/mcdb-103581-notebook-udf-loop branch from 1ad4347 to e3978e2 Compare November 4, 2026 14:15

ShubMishra deployed to Base November 4, 2026 14:15 — with GitHub Actions Active

kesmit13 requested a balanced review from Copilot November 4, 2026 14:36

Copilot started reviewing on behalf of kesmit13 November 4, 2026 14:36

View session

Copilot AI reviewed Nov 4, 2026

View reviewed changes

Merge branch 'main' into skumar/mcdb-103581-notebook-udf-loop

f590ddd

kesmit13 approved these changes Nov 4, 2026

View reviewed changes

kesmit13 deployed to Base November 4, 2026 14:49 — with GitHub Actions Active

kesmit13 merged commit 2a3c5c8 into singlestore-labs:main Nov 4, 2026
13 checks passed

Navigation Menu

Repository navigation

[backend:fix] fix same-kernel Python UDF registration and invocation MCDB-103581 - #156

[backend:fix] fix same-kernel Python UDF registration and invocation MCDB-103581#156
kesmit13 merged 2 commits into
singlestore-labs:mainfrom
ShubMishra:skumar/mcdb-103581-notebook-udf-loop

ShubMishra commented Nov 2, 2026 •

edited by cursor Bot

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

kesmit13 left a comment

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ShubMishra commented Nov 2, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test Plan

Runtime Impact

Deployment Plan

JIRA Issues

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

kesmit13 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ShubMishra commented Nov 2, 2026 •

edited by cursor Bot

Loading